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Abstract. For a cuspidal automorphic representation II of GL(4, A), H. Kim proved that 
the exterior square transfer A 2 II is an isobaric automorphic representation of GL(6, A). In 
this paper we characterize those representations IT for which A 2 II is cuspidal. 



1. Introduction and statement of the main theorem 

Let F be a number field whose adele ring we denote by Ap. Let G\ and G2 be two con- 
nected reductive linear algebraic groups over F and let L G\ and L C?2 be the corresponding 
L- groups. Given an L-homomorphism r : L G\ — * L G2, Langlands principle of functoriality 
predicts the existence of a transfer LT 1— > r(LT) of the L-packet of an automorphic represen- 
tation II of G\(Af) to an L-packet r(n) of automorphic representations of G2(Af). Now 
assume that G2 is a general linear group. We note that an L-packet for a general linear 
group is a singleton set. For applications of functoriality one needs to understand the image 
and fibers of the correspondence II 1— ► r(II). In particular, it is necessary to understand 
what conditions on LT ensure that the transfer r(LT) is cuspidal. 

The main aim of this paper is to describe a cuspidality criterion for the transfer of 
automorphic representations from GL(4,A^) to those of GL(6,Ai?) corresponding to the 
exterior square map A 2 : GL(4, C) — > GL(6, C). Langlands functoriality in this case is a 
deep theorem due to H. Kim [12J. 

Roughly speaking our main result states that if LT is a cuspidal automorphic representa- 
tion of GL(4, Ap), then its exterior square transfer A 2 LT is not cuspidal if and only if LT is 
a transfer of a representation from a 'smaller' group such as GSpin(4), GSp(4), or GL(2) 
over a quadratic extension E/F. It is easy to verify that if LT is a transfer from one such 
'smaller' group, then A 2 II is not cuspidal. It is the converse which is more difficult to prove. 
Another equivalent criterion, which is easier to state, is that A 2 II is not cuspidal if and only 
if LT is essentially self-dual (i.e., LT = LT V <g> x for some Hecke character x of F) or it has a 
nontrivial self-twist (i.e., H = H<g>x fo r a nontrivial Hecke character x of F), except that in 
the essentially self-dual case we need to exclude a particular kind of representation. We are 
not able to satisfactorily explain this exception purely in automorphic terms or in terms of 
L- functions. However, it admits a nice explanation in terms of Galois representations as we 
explain after stating the main theorem. (See also Section [6.41 below.) 
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Let II = <g) v Il v and £ = (gi^E,, be irreducible isobaric automorphic representations of 
GL(4, Af) and GL(6, Af), respectively. Assume that S is a finite set of places of F, includ- 
ing all the archimedean ones, outside of which both of the representations are unramified. 
We say E is an exterior square transfer of II if for all v S we have T> v = A 2 (ILj), i.e., the 
semi-simple conjugacy class in GL(6, C) determining T, v is generated by the image under 
A 2 of the semi-simple conjugacy class in GL(4, C) determining ILj. By strong multiplicity 
one theorem (see Theorem 12.11 below) such a £ would be unique. We will denote it by A 2 II. 
The existence of A 2 II was established by H. Kim [12, Theorem A]. Furthermore, he showed 
that if ILj is not supercuspidal for the places v dividing 2 or 3, then the local component 
U v and (A 2 II) v are compatible via the local Langlands correspondence for GL(4, F v ) and 
GL(6,F 1) ). The assumption at v\2, 3 was made because of complications posed by super- 
cuspidal representations, especially of GL(4, F v ). In any event, it has no bearing on our 
result as we do not need the fact that the local components of A 2 II and II are compatible 
via the local Langlands correspondence at all places. We now state the main theorem of 
this article. 

Theorem 1.1. Let F be a number field and let II be a cuspidal automorphic representation 
of GL(4, Af)- The following are equivalent: 

(i) A 2 II is not cuspidal. 

(ii) II is one of the following: 

(a) II = 7Ti M 7T2, a transfer from GL(2,A j p) x GL(2,A^) to GL(4, A^) via the 
automorphic tensor product M. (This may also be viewed as the transfer from 
the split GSpin(4) to GL(4).J 

(b) II = As(7r), the Asai transfer of a dihedral cuspidal automorphic representation 
ir of GL(2, Ae) where E/F is a quadratic extension. (This may also be viewed 
as the transfer to GL(4) from the quasi-split non-split GSpin*(4) over F which 
splits over E.) 

(c) II is the functorial transfer of a cuspidal representation ir o/GSp(4,A^) asso- 
ciated with the natural embedding of the dual group GSp(4, C) into GL(4, C). 
The representation tt may be taken to be globally generic. 

(d) II = I^(vr) ; the automorphic induction of a cuspidal automorphic representation 
it of GL(2,Ae), where E/F is a quadratic extension. 

(iii) II satisfies one of the following: 

(a) II = II V (g) x for some Hecke character x of F, and II is not the Asai transfer 

of a nondihedral cuspidal representation. 
(/?) II = Il®x f or a nontrivial Hecke character \ of F. 

We now briefly sketch the proof of the theorem. As mentioned above, it is easy to 
verify that (ii) implies both (i) and (iii). In Section [3] we explicitly write down the isobaric 
decomposition of A 2 II for each of the cases (ii) (a)-(ii) (d). In order to check that two isobaric 
representations are isomorphic we repeatedly use a strong multiplicity one theorem, due to 
Jacquet and Shalika, recalled in Section [2.1i 

The proof of (i) (iii), described in Section HI uses some details from the Langlands- 
Shahidi machinery. We show in Proposition 14.11 that if II is not essentially self-dual, then 
A 2 II cannot have a degree 1 or degree 3 isobaric summand. In Proposition 14.21 we verify 
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that if II does not admit a nontrivial self-twist, then A 2 II cannot have a degree 2 isobaric 
summand. For a cuspidal representation a of GL(m, Ajr) consider the Langlands L- function 
L(s, Tlxa, /\ 2 ®p m ), where p m is the standard representation of GL(m, C). These L-functions 
appear in the Langlands-Shahidi machinery for a particular choice of Levi subgroup when 
the ambient group is GSpin(2m + 6). We use Kim [121 §3] to show that, under the above 
mentioned hypothesis on II, these (partial) L-functions are holomorphic at s = 1. The 
implication then follows from a well-known result of Jacquet, Piatetski-Shapiro, and Sha- 
lika recalled in Section 12. 1L We summarize some of the preliminaries we need from the 
Langlands-Shahidi machinery in Section [2.31 

Finally, the proof of (iii) ==>■ (ii) is given in Section [5j It relies on the so-called 'descent 
theory' for classical groups. As a general reference for descent theory we refer to D. Soudry's 
exposition [30] . 

In Section [6] we present a few examples, some only conjectural, illustrating the above 
theorem. In Section 16.41 we ask the question whether it is possible to see the cuspidality 
criterion from the 'Galois side'. The question can be made precise based on the philosophy 
that there is a correspondence between automorphic representations it of GL(n,Ap) and 
^-adic n-dimensional representations a of the absolute Galois group of F. Let us denote this 
correspondence by tt \— > a(ir). Part of this philosophy is that it is supposed to be cuspidal 
if and only if ct(tt) is irreducible. We refer to Ramakrishnan [26] for the state of the art on 
this issue. In view of the above theorem one can ask the following question. Let a be a four- 
dimensional irreducible Galois representation; what condition on a will ensure that A 2 cr is 
irreducible? Upon posing this question in a talk at the Oklahoma Joint Automorphic Forms 
Seminar, A. Kable came up with a very elegant theorem which reflects the equivalence of (i) 
and (iii) in Theorem ll.il We are grateful to him for allowing us to include his theorem and 
its proof in Section T6.41 Recall that in (iii) (a) of Theorem 11.11 above . we had to exclude the 
Asai transfer of a nondihedral cuspidal representation if II is essentially self-dual. On the 
Galois side, this is reflected in the fact that if a four-dimensional irreducible representation 
a is essentially self-dual of orthogonal type, then for A 2 cr to be reducible the image of a 
should lie in the connected component of the identity in the algebraic group GO (4); see 
Theorem 16.51 

Cuspidality criteria are important not only for their intrinsic value in helping us better 
understand a given instance of functoriality but also because they have important arithmetic 
applications. D. Ramakrishnan and S. Wong [28] proved a cuspidality criterion for the 
transfer from GL(2) x GL(3) to GL(6) and used it to construct new cuspidal cohomology 
classes for GL(6). We refer to |21] for a brief survey of cohomological applications of 
Langlands functoriality. H. Kim and F. Shahidi [XT] proved a cuspidality criterion for the 
symmetric fourth transfer from GL(2) to GL(5), which has been used in the study of special 
values of symmetric power L-functions by the second author and F. Shahidi 22] . Such a 
potential arithmetic application was indeed our original motivation to seek a cuspidality 
criterion for the exterior square transfer. 

Acknowledgements: It is a pleasure to thank H. Kim for his encouragement of this project. We also thank 
A. Kable, J.-P. Labesse, and R. Zierau for helpful discussions. This work is partially supported by an 
Alexander von Humboldt Research Fellowship for the first author, and by an ASR +1 grant from the 
College of Arts and Science of the Oklahoma State University for the second author. 
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2. Some preliminaries 

In this section we collect some results we repeatedly use in later sections. To begin, 
we recall a theorem due to Jacquet and Shalika concerning strong multiplicity one for 
isobaric automorphic representations. Then we recall an analytic criterion in terms of 
Rankin-Selberg L-functions, due to Jacquet, Piatetski-Shapiro and Shalika, that charac- 
terizes when two cuspidal automorphic representations are equivalent. Next, we note that 
the natural transfer of automorphic representations of a quasi-split non-split general spin 
group GSpin*(4) to GL(4) is in fact the Asai transfer. Finally, we recall some details from 
the Langlands-Shahidi machinery that will be of use to us, particularly when the ambient 
group is GSpin(m) with m = 8, 10 or 12. 

2.1. Some results of Jacquet, Piatetski-Shapiro, and Shalika. The following strong 
multiplicity one theorem for isobaric representations is due to Jacquet and Shalika |101 [TT] . 

Theorem 2.1. Let tti and tt2 be isobaric automorphic representations of GL(n, Ap). Let 
S be a finite set of places of F , containing the archimedean places, such that both ir\ and 
7T2 are unramified outside S. Ifiri t v — ^2v f or all v £ S, then tt\ = TT2- 

Another useful technical tool for us is the following theorem, due to Jacquet, Piatetski- 
Shapiro and Shalika [9], concerning Rankin-Selberg L-functions. 

Theorem 2.2. Let wi and 1*2 be cuspidal automorphic representations of GL(ni, Ap) and 
GL(n2,Ap), respectively. Let S be a finite set of places containing the archimedean places of 
F and the ramified places ofiri and 7T2 . The partial Rankin-Selberg L-function L s (s, tti x 7^) 
is holomorphic at s = 1 unless n\ = n-i and -K2 = ^\, and in which case it has a simple pole 
at s = 1. 

2.2. The Asai transfer and the quasi-split non-split GSpin*(4). Let E/F be a qua- 
dratic extension of number fields and let V = Tp denote the absolute Galois group of F. 
In this section we let G denote the group GSpin* (4) , a quasi-split non-split linear algebraic 
group over F, which is isomorphic to the split GSpin(4) over E. The L-group of G can be 
written as L G = GSO(4, C) x V, where the Galois action, which factors through G&l(E/F), 
is described below. We note that GSO(4, C) denotes the special orthogonal similitude group; 
one can identify it as a quotient of GL(2,C) x GL(2,C) given by 

GSO(4,C) = /3(GL(2,C) x GL(2,C)), 

where (5 is the map on the right of the exact sequence 

1 — ► C* — ► GL(2,C) x GL(2,C) GO(4,C). 

For details see |251 §2]. Furthermore, the T-action on GSO(4, C) is as follows. If 7 G T and 
g = P(gi,g 2 ) with g { € GL(2, C), then 

= \(3(gi,g2) if i\e = l, 
{P(g2,gi) if7b/f- 

We also need to recall the Asai transfer. Consider the group H = Res^/^GL^) as a 
group over F. Its L-group is given by 

L H = (GL(2,C) x GL(2,C)) x T, 
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with the Galois action given by 

, s j (91,92) if 71^ = 1, 
1 ■{9i,92) = <, , . 

[(52,51) if 71b/ 1 - 

Let W be a 2-dimensional C-vector space, and let V = W ® W. After fixing a basis for 
W, we identify GL(W) with GL(2,C). Consider the map 

As : (GL(W) x GL(W)) x T — ► GL(V) ^ GL(4,C) 

given, on pure tensors, by 

5i6 ® 526 if 7b = 1) 



As(gi,5 2 ;7)(6 ®&) 



5l6®52& if7b/i, 




for all 5; £W and all ^ £ GL(W). It is straightforward to check that this map is indeed a 
homomorphism. It is called the Asai (or 'twisted tensor') homomorphism. (Alternatively, 
one could take the map satisfying As (51,52; 7)(£i ® £2) = —9i£,2 <8>52Ci when 7^ 7^ 1. This 
choice would lead to a quadratic twist of the above map. ) 

Further, let 1 : L G — ► GL(V) = GL(4,C) be the map defined via 

4/3(5i, 52); 7) (6 ®6) 

Again it is straightforward to check that this map is indeed an L-homomorphism. It is now 
clear that t o ((3, id) = As. In other words, the following diagram commutes: 

GL(2, C) x GL(2, C) x T (Aid) ► GSO(4, C) x r 

As 

^GL(4,C) 

Assume that n is a cuspidal automorphic representation of GL(2, A#) and let II = As(7r) 
be its Asai transfer to GL(4, Ap). (See Krishnamurthy [18] or Ramakrishnan [53].) Then 
LT = l(P(tt)), where 0(tt) denotes the transfer of ir to the group GSpin*(4, Ap), and l((3(tt)) 
denotes the transfer of /3(vr) from GSpin*(4, Ap) to GL(4, Ap). The transfer corresponding 
to j3 exists for formal reasons and the existence of the transfer corresponding to t (for generic 
representations) is part of a joint work of the first author with F. Shahidi [21 [3]. 

2.3. The Langlands Shahidi L-functions. Let P = MN be a maximal proper parabolic 
subgroup of a connected reductive quasi-split linear algebraic group G, where M denotes a 
Levi subgroup and N denotes the unipotent radical of P. Let a be a generic automorphic 
representation of M(Ap). Let r denote the adjoint action of the complex Langlands dual 
group M on the Lie algebra of the dual of N. Write r = r± © • • • © r m , where the r»'s 
denote the irreducible constituents of r and the ordering is according to the eigenvalue of 
the adjoint action as in, for example, [29} p. 278]. The Langlands-Shahidi method then 
constructs the L-functions L(s,a,ri) for 1 < i < m. 

We need the following cases of the Langlands-Shahidi method. Let G = GSpin(2n + 6) 
with n = 1,2,3, and consider a maximal parabolic subgroup of G with Levi subgroup 
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M = GL(n) x GSpin(6). (One could also work with split spin groups as in [121 §3]; however, 
we find it more convenient to work with the similitude version of the groups.) The algebraic 
group GSpin(6) is isomorphic to a quotient of GL(1) x Spin(6) by a central subgroup 
A = {1, (—1, c)}, where c is the nontrivial element in the center of Spin(6) of order 2; see [2j 
Proposition 2.2]. The algebraic group GL(4) is isomorphic to a quotient of GL(1) x SL(4) 
by a cyclic central subgroup B of order 4. We identify Spin(6) with SL(4) such that B 
contains A. This way we get a natural map, defined over F, from GSpin(6) to GL(4), which 
in turn induces a map 

(2.3) / : M — ► GL(n) x GL(4). 

Let II be an irreducible cuspidal automorphic representation of GL(4,A^) and let a 
be an irreducible cuspidal automorphic representation of GL(n, Ap), n = 1,2,3. Choose 
any irreducible constituent X of a <g) ~^-\f(M(K F )) an d ^ ^ a l so denote the corresponding 
representation of M(Ap). The Langlands-Shahidi method then gives 

(2.4) L(s, E, n) = L(s, a®n,p n ® A 2 p4), 

where pk denotes the standard representation of GL(k, C), and the L-function on the right- 
hand side is a Langlands L-function; see |12[ §3]. We record a general fact that we need 
from the Langlands-Shahidi method. 

Proposition 2.5. Let wq andwM denote the longest elements of the Weyl group of G and 
M, respectively. Let wq = wqWm- If w${Y?) ^ S, then L(s, £,ri) is entire. 

Proof. This is a standard fact in the Langlands-Shahidi method. For example, see the proof 
of [321 Proposition 3.4]. □ 

In order to apply the above proposition one needs to know the action of wo on a repre- 
sentation of M(Ap). 

Proposition 2.6. Let G = GSpin(2n + 6) with n a positive integer and let a (g> II be a 

representation of M(Ap) as above. Moreover, let wq be as above and denote its image 
under the map \2. 3\) by wq again. Then we have 



wo(a <g> II) 



<t v (g) (II V if n is odd, 

<t v <S> (II ® L0 a ) if n is even. 



Here, uj a denotes the central character of a. 

Proof. Recall that the nontrivial automorphism of the Dynkin diagram of type A m corre- 
sponds to an outer automorphism of GL(m + 1) and it conjugates an irreducible represen- 
tation to its dual representation. The proof of the proposition will follow from a description 
of how wo acts on the root system of type D r . 

We use the Bourbaki notation for the simple roots: 

a\ = E\ — 62, • • • , a r -2 = e r -2 — e r -l> = e r-l — Cr> «r = Cr-l + 

The Weyl group of G is isomorphic with {±l} r_1 x S r which we identify with a subgroup of 
signed r x r permutation matrices acting on W = Rei © • • • © Re r . With this identification 
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observe that 

if r is even, 



wg 




Ir-l 

1 



if r is odd. 



Let r = n+3 and let M be a maximal Levi subgroup of type ^4 n _i x ^3 in G = GSpin(2r). 
The simple roots in the ^3-factor are a r -±,a r -2,a r . The proposition follows by observing 
that 



w (a r -i) = w G w M (a r -i) = w G (-a r ) = w G {-e r ~i-e r 
and 

w (a r ) = WGWM(a r ) = w G (-a r -i) = w G (-€ r -i + e r ) 
while 



I e r _i + e r = a r if r is even, 

1 e r _i — e r = a r _i if r is odd, 

e r _i — e r = a r _i if r is even, 

e r _i + e r = a r if r is odd, 



w (a r - 2 ) = w G w M (a r -2) = w G {-a r - 2 ) = w G {-e r - 2 - e r -i) = «r-2 
in either case. Moreover, for 1 < j < n — 1 we have 

wo(aj) = ^G^M(ai) = ^G(-an-i) = w G (-e n -j + e n _ J+ i) = a n -j. 

This means that tt?o induces the nontrivial automorphism of the Dynkin diagram of the 
A n _i-factor of M, and on the ^-factor it induces the nontrivial automorphism of the 
Dynkin diagram if and only if r is even. 

Let m = m(g, h) be an arbitrary element in the Levi subgroup M identified with GL(n) x 
GSpin(6), in G = GSpin(2n + 6), where g G GL(ra) and h G GSpin(6), and let v = v(m) 
denote its similitude character value. Then 

w m(g,h)wQ 1 = m^g 1 u(m),h*), 

where 

j t h 1 if r is even, 



h 

We conclude that 



if r is odd. 



w ((T<S)U)(m(g,h)) = (a ® II) (m(*p 1 v(m),h*)) 

= aCg-'Mh^iuim)) 

= ^(g)U*(h)u a (u(h)) 

= (a v (n* ®w a )) (m(g,h)), 



where 

n 



I n v if r is even, 
III if r is odd. 

This completes the proof. Note that r = n + 3 is even if and only if n is odd. □ 
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3. The proof of (ii)=>(i) 

We verify that for each of (ii)(a) through (ii)(d) the exterior square transfer A 2 II is not 
cuspidal. Indeed, it is not difficult to write down the isobaric decomposition for A 2 il in 
each case. 

3.1. (ii)(a)=Ki). 

Proposition 3.1. Let tt\ and 7T2 be cuspidal automorphic representations o/GL(2,A^). 
Let 7ri K1-7T2 be the transfer to an automorphic representation o/GL(4, Ap), whose existence 
was established in [23]. For brevity, we let U = n\ M 7r 2 and uj = u> ni u> n2 . We have 

(a) A 2 (vri m vr 2 ) = (Sym 2 (vri) ® L0 n2 ) ffl (Sym 2 (vr 2 ) ® u Vl ) . 

(b) Assuming Langlands functoriality one should expect 

Sym 2 (7ri M tt 2 ) = (Sym 2 (^) H Sym 2 (7r 2 )) ffl w^w^. 

(c) The partial L-function L s (s, A 2 (I1) ® entire while the partial L-function 
L s (s, II, Sym 2 ® /ias a pole at s = 1. 

Proof. The proof of (a) and (b), using Theorem I2.1|. is an easy calculation using Satake 
parameters on both sides. More precisely, for a finite place v at which both tt\ and 7r 2 
are unramified we let n\ v and 7r 2j „ have Frobenius-Hecke eigenvalues t± = diag(ai,&i) and 
t 2 = diag(o 2 ,6 2 ), respectively. Then 

A 2 (ii ® i 2 ) = (diag(a 2 , ai&i, 6 2 ) ■ a 2 6 2 ) EH (diag^, a 2 6 2 , 6 2 ) • ai&i) 

and 

Sym 2 (ti ® i 2 ) = (diag(a 2 , ai6i, &f) ® diag(o 2 , a 2 6 2 , 6 2 )) EE (ai&i • o 2 6 2 ) . 

Part (a) has also been observed by others; see [HI (7.27)] and [271 (2-6)], for example. 
For (b) to make sense one has to assume the symmetric square transfer from GL(4) to 
GL(10) and the automorphic tensor product from GL(3) x GL(3) to GL(9), both particular 
instances of functoriality. 

To prove (c), observe that il v = II ® which implies 

L S { S , n x n v ) = L S ( S , A 2 (n) ® uj- 1 )l s { s , n, s ym 2 ® w" 1 ), 

where S is a finite set of places including all the archimedean ones such that II is unramified 
outside of S. From (a) we have A^n)®^ -1 = Ad(7ri)EBAd(7r 2 ). (Here Ad(7Tj) = Sym 2 (7Tj)<g> 
u>~ .) If 7Tj is not dihedral, then Ad(7Tj) is cuspidal (by Gelbart- Jacquet [7]) and hence its 
partial L-function is entire. If 7Tj is dihedral, say 7Tj = Ip{x)i then it is easy to see that 
Ad(-7Tj) = uje/f ffl ^(x'x -1 )) where oje/f is the quadratic character of F associated to E 
by class field theory, and x' is t ne nontrivial Gal(£'/F)-conjugate of %■ Since 7Tj is cuspidal, 
the inducing character \ is Galois regular, i.e., x' 7^ or equivalently x'x 1 is a nontrivial 
character, whence L s (s, Ad(7Tj)) = L s {s,uje/f)L s {s,L^{x'x^ 1 )) is entire. (In particular, it 
does not have a pole at s = 1.) Therefore L 5 (s, A 2 (II) ®w _1 ) = L 5 (s, Ad(7Ti))L s '(s, Ad(7r 2 )) 
does not have a pole at s = 1. However, L s (s,n x n v ) has a pole at s = 1, which implies 
that L (s, n, Sym 2 ® has a pole at s = 1. 

Note that (c), unlike (b), is unconditional and does not depend on assuming unproven 
instances of functoriality. □ 
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3.2. (ii)(b)=Ki). 

Proposition 3.2. Let E/F be a quadratic extension. Let tt be a cuspidal automorphic 
representation of GL(2, Ae) and let II = As(7r) be its Asai transfer. Assume that II is a 
cuspidal automorphic representation o/GL(4, A_p). Then A 2 II is cuspidal if and only if tt 
is not dihedral. 

Proof. The proof depends on the following identity 

A 2 (As(tt)) =lf(Sym 2 7r®u4), 

where ' means the nontrivial Gal(£'/F)-conjugate. (See Krishnamurthy |X8|, §7].) To begin, 
assume that tt is not dihedral. By Ramakrishnan [ 251 Theorem 1.4] we know that As(7r) is 
cuspidal if and only if tt' ^ tt ® fi for any fi. If A 2 II is not cuspidal, then 

(Sym 2 vr <g> uj^)' = Sym 2 vr <g> uj' n . 

This implies that Sym 2 7r' <£) uj w = Sym 2 7r (g) uj'^, i.e., Ad(7r) = Ad(7r'). This, in turn, implies 
that tt' = tt ® fj, (by Ramakrishnan |23^ Theorem 4.1.2]), contradicting the fact that there 
is no such twist. Hence A 2 I1 is cuspidal. 

Next, assume that tt is dihedral. In this case Sym 2 (7r) is not cuspidal and therefore, 
I^(Sym 2 7r © cannot possibly be cuspidal. □ 

3.3. (ii)(c)^(i). 

Proposition 3.3. LetH be a cuspidal automorphic representation o/GL(4, Aj?) and assume 
that II is a transfer from a cuspidal (generic) automorphic representation tt of GSp(4, Ap). 
Then 

A 2 II = f 5 (7r) fflw,, 

where F5 is a degree 5 representation of GSp(4, C) defined below. In particular, A 2 II is not 
cuspidal. 

Proof. We use Kim [131 P- 2793]. As observed there, one has 

Sp(4,C) w GL(4,C) GL(6,C) 

and A 2 oi = r§ © 1 decomposes into a direct sum of the trivial representation and a 
five-dimensional representation r§. Similarly, 

GSp(4,C) ^ GL(4,C) GL(6,C) 

and A 2 o Z = © v, where v is the similitude character of GSp(4, C) and is a five- 
dimensional representation of GSp(4, C). This implies the desired equality of automorphic 
representations. □ 

Remark 3.4. Embedded in the above proof is the assertion that a cuspidal (generic) auto- 
morphic representation tt of GSp(4, Ap) admits a transfer to an automorphic representation 
7% (71") of GLs(Air) corresponding to the representation r^. This depends on the generic trans- 
fer from GSp(4) to GL(4) (see Asgari-Shahidi [1]), and the exterior square transfer from 
GL(4) to GL(6) due to Kim p]. 
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3.4. (ii)(d)=Ki). 

Proposition 3.5. Let ir be a cuspidal automorphic representation of GL(2,Ae), where 
E/F is a quadratic extension, and let II = lfi(7r) be the automorphic induction of it to an 
automorphic representation o/GL(4, Ap). Then A 2 II is not cuspidal. 

Proof. It is known that 

A 2 (lf(vr)) = As(tt)®uj e/f ffl l£K), 

where oje/f is the quadratic Hecke character of F associated to E/F by class field theory. 
See, for example, Kim [T5l §3]. □ 

4. The proof of (i)=>(iii) 

It is equivalent to prove that if II is a cuspidal automorphic representation of GL(4, Ap) 
which neither has a nontrivial self- twist nor is essentially self-dual, then A 2 (II) is cuspidal. 
Observe that if an isobaric automorphic representation p of GL(6,Ap) is not cuspidal, 
then it must have an isobaric summand of degree 1,2, or 3, i.e., there exists a cuspidal 
representation a of GL(n, Ap), with 1 < n < 3, such that L s (s,p x a) has a pole at s = 1. 
Again S denotes a finite set of places of F, including all the archimedean ones, such that 
all the representations involved are unramified at places outside S. Now with II as above, 
the cuspidality of A 2 II follows from the following two propositions. 

Proposition 4.1. If U ^ II V ® X f or a M X; then L s (s,H ® a, A 2 ® P2) is holomorphic at 
s = 1 for every cuspidal representation a o/GL(n, Ap), n = 1,3. 

Proof. Let £ be as in (|2.3|) . By Proposition 12.51 it is enough to show that wo(T,) ^ S. If 
we have wo(E) = S, then wo(a ® II) = a ® II. On the other hand, by Proposition 12.61 we 
have wo(a <8> II) = cr v <8> (II V (8) In particular, we must have II = il v (g) tJo- contradicting 
the hypothesis. □ 

Proposition 4.2. //IT ^ II®x / or a ^ nontrivial x, then L s (s, IL<g>o~, A 2 ®^) is holomorphic 
at s = 1 /or every cuspidal representation a of GL(2,Ap). 

Proof. The same argument as in the above proof works as long as w ff / 1 because by 
Proposition 12.61 we have wo(a ® II) = o" v ® (II ® uv) ^ a <g>TL. This means that if a is a 
cuspidal representation of GL(2, Ap) with nontrivial central character, then a cannot occur 
as an isobaric summand of A 2 (II). 

Now suppose that a is a cuspidal representation of GL(2,A^) with trivial central char- 
acter and that a is an isobaric summand of A 2 (II). Then the representation a ® 6 2 occurs 
in A 2 (II ® 9) for any Hecke character 9. Note that n ® 9 also satisfies the hypothesis that 
it has no nontrivial self-twists. Choose 9 such that 

u^ 6 2 = u a 9 4 = 9 4 ^1 

to get a contradiction. □ 

5. The proof of (iii)=^(ii) 

We prove that if n satisfies (iii)(/3), then it is of the form (ii)(d), and if it satisfies 
(iii)(a), then it is one of (ii)(a)-(c). 
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5.1. (iii)(/3) => (ii)(d). Assume that 

(5.1) U^U^x 

for some nontrivial x- Taking central characters we have x 4 = 1- If X 2 7^ H> then we 
may replace x with x 2 m (|5.ip . which means we may assume that the character x m 
(|5.ip is quadratic. We want to show that II is induced from a quadratic extension. If II 
is a representation of GL(2), then the analogous statement is a well-known result due to 
Labesse-Langlands |19j . In our case it follows from the work of Arthur-Clozel pQ and some 
L-function arguments as we explain below. 

Lemma 5.2. Let II be a cuspidal representation of GL(2ra, Ap) satisfying II = II <g> x for 
a nontrivial quadratic character x- Then IT = I^(vr), where E/F is the quadratic extension 
associated with x an d it is a cuspidal representation o/GL(n, A#). 

Proof. We first claim that the base change He is not cuspidal. To see this, assume that it 
is cuspidal. For a finite set S of places of F and the corresponding set T of places of E 
lying above those in S, as before, we have 

(5.3) L T (i,n E x^) = L s { s ,u x n v )L s (s,n x n v ® x ) 

= L s (s,UxU v ) 2 . 

For sufficiently large S, the left hand side of (|5.3|) has a simple pole at s = 1 while the right 
hand side has a double pole at s = 1. This contradiction shows that IL3 is not cuspidal. 
This means that He = 7riffl-7r2, where 7Tj are cuspidal representations of GL(n, A#). Further, 
7Ti 7T2, because if they are equivalent, then 

L 5 (s,n x n v ) 2 = L T {s,U E x n^) = L T (s,7Ti x vr^) 4 , 

but L (s,n x II V ) 2 has a double pole and L T (s,iri x 7r^) 4 has a pole of order 4 at s = 1. 

Next, we claim that II = lfJ(7Ti). To show this it is enough to prove that the partial 
L-function L s (s,I^(tti) x II v ) has a simple pole at s = 1. This follows from 

L 5 M£K)xn v ) = L 5 M£Kxn)0) 

= L t (s,tt 1 xU e ) 

= L T (s,TTi X TTi)L T (s,1Ti X TT^)- 

Since TT2 ^= ir\ we know that L T (s, tti x tt^)L t (s, m x -zr^ ) has a simple pole at s = 1. □ 

5.2. (iii)(a) ==> (ii)(a)— (c). Now assume that 

(5.4) n s n v x 

for some x- For a finite set S of places of L, as before, we have 

L 5 (s,nxn v ) = l s (s,u x (n®x -1 )) 
= L^fnxn)^- 1 ) 

= L s (s, n, A 2 ® x _1 )i S (s, n, Sym 2 x" 1 )- 

The last two L-functions are the standard twisted exterior square and twisted symmetric 
square L-functions of II. If S is sufficiently large, then L s (s,H x II V ) has a simple pole at 
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s = 1. Therefore one and exactly one of the partial twisted exterior or symmetric square 
L- functions has a simple pole at s = 1. 

First, assume that L s (s,H, A 2 <g> x _1 ) has a pole at s = 1. Then there exists a cuspidal 
representation ir, which may be taken to be globally generic, of GSp(4, A^) such that II is 
the functorial transfer of II, i.e., II is a representation as in (ii)(c). This result has been 
known for a long time and, we believe, is originally due to Jacquet, Piatetski-Shapiro, and 
Shalika. See Gan-Takeda [6] for a proof. It would also follow from the more general method 
of "descent" as we explain below. 

Next, assume that L s (s, II, Sym 2 <S>X -1 ) has a pole at s = 1. Taking central characters 
in (15. 4ft we have wn = u^x 4 - In other words, /i = tJnX -2 is a quadratic character. If /i is 
trivial, then II is a transfer from a cuspidal representation ir of GSpin(4), a split connected 
reductive group of type -D4 whose derived group is Spin(4). If /u is nontrivial, then II is 
a functorial transfer from the quasi-split non-split group GSpin*(4) associated with the 
quadratic extension E/F attached to [i. 

These facts can be proved using the "descent" method of Ginzburg-Rallis-Soudry. If x 
is trivial, then II would be a transfer from a special orthogonal or symplectic group. We 
refer to Ginzburg-Rallis-Soudry [8j Theorem A] and Soudry [30 1 Theorem 4 and 12] for 
the proofs for classical groups. For the more general case of classical similitude groups we 
understand that J. Hundley and E. Sayag are now in the process of writing down a proof 
of this fact among other things. For certitude, we state the precise statement we are using 
from the descent theory below. 

Let II be a cuspidal automorphic representation of GL(4, Ap) satisfying II = 
II V <g>x- Assume that the partial L-function L s (s, II, Sym 2 <S>x -1 ) has a pole 
at s = 1 for a sufficiently large finite set S of places of F. If fi = luuX~ 2 
is trivial, then there exists a globally generic cuspidal representation ir of 
GSpin(4,A^) such that ir transfers to II. If fj, is a nontrivial (necessarily 
quadratic) character, and E/F is the associated quadratic extension, then 
there exists a globally generic cuspidal representation n of GSpin*(4, Ap) 
such that 7r transfers to II. (In our notation GSpin*(4) is a non-split quasi- 
split group over F which is isomorphic to the split GSpin(4) over E.) 

With the above notation, if \i is trivial, then II is as in (ii)(a), and if it is nontrivial then 
II is as in (ii)(b). 

6. Examples and Complements 

In this section we give a few examples of our main result. In some of them the proposed 
representation II of GL(4, Ap) is not yet proved to be automorphic, but it is conjecturally so. 
We also comment on possible intersection among the four cases in part (ii) of Theorem ll.il 
Finally, we present a theorem due to A. Kable on when the exterior square of an irreducible 
four-dimensional representation is reducible. 

6.1. K. Martin's G192. The matrices 
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in GL(4, C) generate a group G192 of order 192. Let p be the four-dimensional representa- 
tion of the group G192 given by inclusion. Then p is an irreducible representation. In [20] 
K. Martin showed that p is modular, i.e., there exists a (cuspidal) automorphic representa- 
tion II(p) that corresponds to p. 

Example 6.1. Let IT = II (p). Then II 1 is a cuspidal automorphic representation of 
GL(4, Ap). Moreover, it is neither essentially self-dual nor does it have a nontrivial self- 
twist. It is not on the list of possibilities of (ii). Furthermore, A 2 IIi is cuspidal. In other 
words, LTi is an example of a cuspidal representation which does not satisfy any of (i)-(iii) 
of Theorem 11.11 



Proof. First we check that /\ 2 (p) is an irreducible representation. To see this consider the 



standard basis (ex, e 2 , e 3 , e 4 ) for 
given by 



I and fix the ordered basis (w\,W2, ■ ■ ■ , wq) of C 6 = A 



2,p4 



w\ = e\ ® e2 — e 2 ® ex, itf 2 = ex ® e 3 - e 3 ® ex, w 3 = ex ® e 4 - e 4 (£> ex, 
W4 = e 2 e 3 - e 3 <g> e 2 , w 5 = e 2 <8> e 4 - e 4 ® e 2 , w 6 = e 3 <8> e 4 - e 4 (8» e 3 . 

Let A,B,C,D be the images of a,b,c,d under A 2 , respectively. Then, with respect to the 
above basis, we have 

"l 

-1 



A 



-1 



-1 



B 



C 



1 



-1 



1 



X 



It is easy to check that if a 6 x 6 matrix X commutes with A, B, and C, then it has to be 
of the form 

a b 
c 

e f 
-f e 

d 

—b a 

Further, if XD = DX, then a = c = d = e and b = f = 0, i.e., X is a scalar matrix. There- 
fore, Hoiiig 192 (A 2 p, A 2 p) = C and, by Schur's lemma, the representation A 2 p is irreducible. 
This implies that A 2 Ili is cuspidal. (This is because if a complex Galois representation a 
is modular, i.e., corresponds to an automorphic representation tt = ir(a), then a is irre- 
ducible if and only if tt is cuspidal. This fact follows from L(s, a % <r v ) = L(s, tt x 7r v ); see 
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Ramakrishnan [261 Introduction].) Martin observes that p, and hence III, is not essentially 
self-dual. Clearly III is not on the list of possibilities in (ii) of the main theorem because 
if it were, then A 2 IIi would not be cuspidal (see Section [3]). Hence 111 does not satisfy any 
of the equivalent statements of Theorem 11.11 □ 

In [20] Martin considers a four-dimensional irreducible representation p of the absolute 
Galois group of Q whose image in PGL(4, C), denoted G, is an extension of A4 by V4. In this 
situation G is either V4 x A4 or V4 • A4. In the former case p is of GO(4)-type and II = IL(p) 
is a transfer from GL(2, A_p) x GL(2, Ap), which is contained in our (ii)(a). The example 
of G192 is an instance of the latter situation. In either case, H(p) may also be thought of 
as being obtained by automorphic induction across a non-normal quartic extension with no 
quadratic subextension. 

6.2. The standard representation of S5. Consider a tower of number fields J/J/E/F. 
Here, E/F and J/ J are quadratic extensions, J/E is an ^-extension, J/F is an S5- 
extension, J/E is an SL(2, F5)-extension, and J/F is Galois. (Recall that A§ = PSL(2,F5).) 
In what follows we identify A§, S5 and SL(2,Fs) with these Galois groups. 
Let a be the standard four-dimensional irreducible representation of S$, 

a: S 5 — ► GL(4,C). 

Here are some properties of a: 

(1) A 2 cj is a six-dimensional irreducible representation of S5 (see [H §3.2]). 

(2) cf\a 5 is irreducible (because a 7^ a ® e, where e is the sign character of S5). 

(3) A 2 (cr|yi 5 ) = (A 2 ct)|a 5 is reducible (because A 2 (cr) = A 2 (<r) <g) e), and its irreducible 
constituents are both of degree 3. 

(4) a is self-dual (because the character of a has integer values). 

(5) ct\a 5 = pi <8> P2, where p\ and P2 are the two-dimensional irreducible representations 
of SL(2,F 5 ) (see [H Lemma 5.1]). 

We need some details about the pi. Let p be the unique (up to twists) cuspidal representation 
of GL(2,Fs) whose restriction to SL(2,Fs) is reducible. In this case, /5|sl(2,f 5 ) = Pi® Pi- If 
g G GL(2,F 5 ) - Z(F 5 )SL(2,F 5 ), then p{ = p 2 . Here Z is the center of GL(2). Conjugating 
SL(2,Fs) by such an element g induces the nontrivial outer automorphism of SL(2,Fs) 
because if it were an inner automorphism, then we would have p\ = p%, which contradicts 
the fact that the restriction from GL2 to SL2 is multiplicity free. 

Let p denote either p\ or p 2 . In constructing examples (to illustrate our main theorem), 
we make the following assumption: p is modular, i.e., there exists a cuspidal automorphic 
representation vr(p) of GL(2, Ae) with p <-> n(p). In this situation, it is expected |T6| that 
there exists an automorphic representation vr(cr) of GL(4, Ap) with n(o~) <-» a and 7r(<r) is 
the Asai transfer of 7r(p), i.e., vr(cr) = As (n(p)). 

Example 6.2. Let n2 = 7r(<r) = As(7r(p)). Then n2 is a cuspidal automorphic represen- 
tation of GL(4, Ap) which is self-dual. However, it is the Asai transfer of a nondihedral 
cuspidal representation. Moreover, it has no nontrivial self-twists and it is not on the list 
of possibilities in (ii). Furthermore, its exterior square transfer A 2 n2 is cuspidal. In other 
words, n2 is an example of a cuspidal representation which does not satisfy any of (i)-(iii) 
of Theorem 11.11 
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Proof. Since II2 = n(a) and a is irreducible, we conclude that II2 is cuspidal. (Cuspidality 
of II2 may also be seen by appealing to the cuspidality criterion for the Asai transfer due 
to Ramakrishnan [25} Theorem 1.4].) 

Next, we note that II2 is self-dual and is the Asai transfer of a cuspidal representation, 
namely ir(p). Note that tt(p) is not dihedral as p is not induced from a character of an index 
two subgroup because there is no such subgroup in SL(2,Fs). Also, 1I2 has no nontrivial 
self- twists because a has no nontrivial self- twists. 

Finally, note that A 2 Il2 is cuspidal since A 2 Il2 = A 2 7r(<j) = 7t(A 2 ct) and A 2 <r is an irre- 
ducible representation implying that 7r(A 2 o") is cuspidal. (See, for example, Ramakrishnan 
[261 Introduction].) □ 

Example 6.3. Let II3 = (n 2 )^ be the base change of H2 to an automorphic representation 
of GL(4, Ae)- Then II3 is a cuspidal automorphic representation of GL(4, A#) which is self- 
dual and is not the Asai transfer of a nondihedral representation. It is contained in (ii)(a) 
and its exterior square transfer A 2 H3 is not cuspidal. In other words, II3 is an example of 
a cuspidal representation which satisfies (i)-(iii) of Theorem I l.li 

Proof. To see cuspidality of II3, as well as the fact that it is contained in (ii)(a), note that 

n 3 = (n 2 )£ = n(o-) E = ttOUs) = 7r(pi ® p 2 ) = 7r(pi) M 7t(/9 2 ). 

Neither pi is monomial since SL(2,Fs) does not have an index two subgroup. Applying 
the cuspidality criterion for ir(pi) M 7r(p 2 ) due to Ramakrishnan [24} Theorem 11.1], we 
see that II3 is not cuspidal if and only if tt(pi) = ir(p 2 ) <8> p for some Hecke character p 
of E. On the other hand, 7r(/? 2 ) <8> p = n{p2 ® p), where we identify the Hecke character 
p with a character of the absolute Galois group of E via global class field theory. Hence, 
we have ir(pi) = 7r(p 2 ® p)- This implies that p\ = p 2 <S> p (since, for any two Galois 
representations t\ and t 2 , one has tt{t{) = 7r(r 2 ) if and only if t\ = t 2 ; one can see this by 
considering the equality L s (s, 7r(ri) x 7r(r2) v ) = L s (s, t\ ® r^)). Therefore, p is a character 
of Gal(J/E) = SL(2,Fs), a perfect group, hence p is trivial. Whence p\ = P2, which 
contradicts the fact that they are inequivalent as was observed earlier. 
Next, observe that A 2 n3 is not cuspidal because 

A 2 n 3 = A 2 (vr( / o 1 ) M7r(p 2 )) = (Sym 2 (vr( / o 1 )) ® u w(p2 )) © (Sym 2 (^(/) 2 )) «, w ), 

which is of isobaric type (3,3). (See Proposition 13.11 ) 

Finally, we observe that n3 is self-dual because cr, and hence o\a 5 i is self-dual and that 
n3 could not be an Asai transfer of a nondihedral representation because if it were, then 
A 2 (n3) would be cuspidal by Proposition 13.21 □ 

6.3. On possible intersections between representations in (ii). The purpose of this 
subsection is to show that the cases (ii)(a) through (ii) (d) are not mutually exclusive. 

Example 6.4. Let tt = ifj(x) ^ e a cus pidal automorphic representation of GL(2, Ap) which 
is automorphically induced from a Hecke character x of -E, where E/F is a quadratic 
extension. Let r be a nondihedral cuspidal automorphic representation of GL(2,A^). Let 
H4 = ttMt. Then n4 is a representation that is common to (ii)(a), (ii) (b) and (ii) (c) . 
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Proof. Prom Ramakrishnan's cuspidality criterion [241 Theorem 11.1] we know that H4 is a 
cuspidal representation of GL(4, Af)- By construction, II4 is in (ii)(a). 
We observe that 

Yl A = I F E { X )^T = l F E {X®T E ). 

Since the induced representation l E (x <8 te) is cuspidal, the inducing representation x® T E 
is, a fortiori, cuspidal. Hence II4 is in (ii)(b). 

Now we claim that II is also a transfer from a (generic) cuspidal representation of 
GSp(4, Af)- To see this we recall the following well known identities: 

Sym 2 (l£( X )) = l£( X 2 )ffl X | A x, 

a 2 (Ie(x)) = x\ a *- u e/f, 

where oje/f is the quadratic Hecke character of F associated to E/F by class field theory. 
In particular, the central character of ir is given by u> n = x\& x '^e/f- From Proposition 13. II 
we have 

A 2 (vr M r) = (Sym 2 (vr) (g> uj t ) ffl (Sym 2 (r) ® u%) . 
For brevity write u> = uj^uJt-. We deduce that 

A 2 (n 4 ) = {Ie(x 2 ) ® w r ) ffl ujuje/f ffl (Sym 2 (r) g> u n ) . 

Hence, the partial L-function L s \s,H, A 2 <8> {lolo e / f )~ v ) has a pole at s = 1. Applying a 
recent result of Gan and Takeda [6] we conclude that n is a transfer from GSp(4), i.e., n is 
in (ii)(c). □ 

6.4. A calculation on the Galois side. As mentioned in the introduction, one may ask 
for an irreducibility criterion on the Galois side, i.e., for £-adic Galois representations, which 
reflects the cuspidality criterion one is looking for. In this section we present such a theorem 
due to A. Kable. We are grateful to him for the permission to include this material here. 
Theorem 16.51 below is the analogue of the equivalence of (i) and (iii) in Theorem 11.11 We 
begin by reviewing some preliminaries. 

Let k be an algebraically closed field whose characteristic is not two and let V be a 
four-dimensional A:-vector space. Fix a nonzero element rj G A 4 V. There is a nondegenerate 
symmetric bilinear form B on A 2 ^ defined by u\ A u>2 = B(oji,oj2)v for all U\,U2 G A 2 V. 
The bilinear space (A 2 ^, B) is isomorphic to the orthogonal sum of three hyperbolic planes. 
Let GO(.B) be the group of similitudes of B, A : GO(-B) — > k x the similitude character, and 
GSO(£?) the subgroup of proper similitudes. This subgroup consists of those T G GO(B) 
such that det(T) = A(T) 3 , and it coincides with the connected component of the identity 
in the algebraic group GO(-B). (In the literature, the group GSO(B) is also denoted by 
SGO(-B) or GO + (-B).) We use similar notation also for the similitude groups of forms on 
V itself. Let p : GL(V) -> GL(A 2 V) be the homomorphism p(S) = A 2 S. 

Let G be a group and let a be an irreducible representation of G on V. Recall that a is 
essentially self-dual if there is a character x of G such that <r v = x ® cr. In this case, x ls 
a subrepresentation of a <g) a. We say that a has symplectic type if x _1 occurs in A 2 cj and 
orthogonal type if x _1 occurs in Sym 2 <7. If a is essentially self-dual of orthogonal type, then 
there is a nonzero symmetric bilinear form C on V such that G acts on V by similitudes 
of C. The kernel of C is a G-invariant proper subspace of V and hence trivial. Thus, C 
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is nondegenerate and cr(G) C GO(C). If a(G) C GSO(C), then we say that a is of proper 
orthogonal type; otherwise, we say that a is of improper orthogonal type. Finally, we say 
that a has a nontrivial quadratic self-twist if there is a nontrivial {±l}-valued character \ 
of G such that a = x ® 

Theorem 6.5 (A. Kable). Let a be an irreducible A- dimensional representation of a group 
G over an algebraically closed field whose characteristic is not two. Then the following two 
conditions on a are equivalent: 

(1) A 2 cr is reducible. 

(2) a satisfies at least one of the following: 

(a) is essentially self-dual of symplectic type, 

(b) has a nontrivial quadratic self-twist, or 

(c) is essentially self-dual of proper orthogonal type. 

Toward the proof of the above theorem, we begin with a lemma. 

Lemma 6.6. The bilinear space (A 2 V, B) has the following properties. 

(1) The image of p is GSO(-B). 

(2) An isotropic line in A 2 V has the form A 2 Q, where Q < V is a uniquely determined 
2-dimensional subspace ofV. 

(3) An isotropic 3-space in A 2 V either has the form LAV, where L < V is a uniquely 
determined line, or the form A 2 U, where U < V is a uniquely determined 3-space. 

(4) Let W < A 2 V be a 3-space on which B is nondegenerate. Then there is a nonde- 
generate symmetric bilinear form C on V such that 



The form C is determined by W up to scalars. Every nondegenerate symmetric 
bilinear form on V occurs in this way for a suitable choice ofW. 

Proof. We omit the proofs of (1), (2), and (3), as they are easy exercises, and briefly sketch 
the proof of (4). Let Q be the quadric hypersurface in P(A 2 V) consisting of null vectors 
for B. By (2), we may identify Q with the Grassmannian of lines in P(V). Let W be a 
3-dimensional subspace of A 2 V on which B is nondegenerate, and Y = P(W) n Q be the 
smooth plane conic defined by B\yy- Let T be the subvariety of P(V) obtained by taking 
the union of the lines in P(V) corresponding to points of Y. It is easily verified that T is 
a smooth quadric hypersurface, and so there is a nondegenerate symmetric bilinear form C 
on V, unique up to scalars, such that T has equation C(v, v) = 0. An element g G GL(V) 
preserves T together with the ruling T — > Y sending a line in T to the corresponding point 
of Y if and only if g £ GSO(C). From the construction, the set of g with this property is 
the same as the set of all g such that p(g)(W) = W . The last claim in (4) follows from the 
fact that GL(F) acts transitively on the set of all nondegenerate symmetric bilinear forms 



Proposition 6.7. The representation (a, V) is essentially self-dual of symplectic type if 
and only if A 2 V contains a G-invariant line. 



{g G GL(V) | p(g)(W) = W} = GSO(C). 



on V. 



□ 



Proof. This follows immediately from the definitions. 



□ 
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Lemma 6.8. There is no G-invariant isotropic 3-space in A 2 V. 

Proof. If there is such a 3-space, then, by Lemma |6.6| it is of the form LAV or of the form 
A 2 U. Note that G-invariance of the 3-space combined with the uniqueness statements in 
Lemma 16.61 imply that L or U is also G-invariant, which contradicts irreducibility of a. □ 

The proof of the following proposition would be substantially simpler if the action of G 
on A 2 V were completely reducible. However, in the current generality this need not be true. 

Proposition 6.9. Suppose that a is not essentially self-dual of symplectic type. Then a 
has a nontrivial quadratic self-twist if and only if A 2 V contains a G-invariant 2-space. 

Proof. Suppose first that a has a nontrivial quadratic self-twist, say by the character x- 
Let H be the kernel of x an d recall that, by Clifford theory, o~\h is the sum of two 2- 
dimensional subrepresentations. Let W < V be the P-invariant 2-space on which one of 
these subrepresentations is realized. Then it is easy to see that the G-translates of A 2 W 
span a G-invariant 2-space in A 2 V . (The reader should compare this with the proof of 
Proposition 13.51 Indeed, the 2-dimensional G-invariant subspace is the induction to G of 
the determinant character of the representation of H on W. Recall that if a is a Galois 
representation that corresponds to an automorphic representation it, then the determinant 
character of a corresponds to the central character of it.) 

Now suppose that A 2 V contains a G-invariant 2-space P. The kernel of B\p is G-invariant 
and thus is either {0} or P, for the first hypothesis implies that there can be no G-invariant 
line in A 2 V . Suppose that the kernel is P. Then the 4-space P 1 - contains P and the form B 
and the action of G pass down to P^/P. Suppose that B has a nontrivial kernel in P^/P. 
This kernel cannot be all of P /P, for then P 1 - would be an isotropic 4-space in A 2 V. 
Thus the kernel must be a line in P^/P and this kernel is necessarily G-invariant. The 
preimage of this line in P 1 - is an isotropic G-invariant 3-space in A 2 V, contrary to Lemma 
16.81 We conclude that (P ± /P,B) is a nondegenerate quadratic 2-space. Such a space is 
isomorphic to a hyperbolic plane and hence contains exactly two isotropic lines. The action 
of G on P^/P is by similitudes, hence it permutes these lines. Taking the preimage in P , 
we obtain two isotropic 3-spaces Aj and A2 in A 2 V that are permuted by G. By Lemma 
16.81 these isotropic G-spaces cannot be fixed by G and we conclude that the stabilizer of 
each is a subgroup H of index two in G. By repeating the argument of Lemma 16.81 with 
H in place of G, we conclude that there is either a line L < V or a 3-space U < V that is 
//-invariant. By replacing a by <r v if necessary, we may assume that the former possibility 
holds. Let go £ G — H. Then the 2-space L + a{go)L is easily seen to be G-invariant, 
contrary to the irreducibility of a and cr v . This contradiction finally allows us to conclude 
that the restriction of B to P is nondegenerate. 

We now repeat the argument of the previous paragraph with the space P in place of the 
space P^/P. It yields an index two subgroup H of G and two isotropic lines in A 2 V that 
are fixed by H. By Lemma [6.61 each of these lines has the form A 2 Q with Q < V a 2-space. 
By the uniqueness assertion from Lemma 16.61 each of these 2-spaces is PT-invariant. It now 
follows from Clifford theory that if x is th e nontrivial {±l}-valued character on G whose 
kernel is H, then x ® o = a. Thus a has a nontrivial quadratic self-twist, as required. □ 
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Proposition 6.10. Suppose that a is neither essentially self-dual of symplectic type nor 
has a nontrivial quadratic self-twist. Then a is essentially self-dual of proper orthogonal 
type if and only if A 2 V contains a G-invariant 3-space. 

Proof. Let W < A 2 V be a G-invariant 3-space. By Lemma l6.8l W cannot be isotropic. The 
group G acts on W by similitudes and so the kernel of B\w is G-invariant. We have just 
observed that this kernel cannot be W and, by the hypotheses and the preceding results, it 
cannot be of dimension 1 or 2. Thus the restriction of B to W is nondegenerate. It follows 
from Lemma 16.61 that there is a nondegenerate symmetric bilinear form C on V such that 
cr(G) C GSO(G). This implies that a is essentially self-dual of proper orthogonal type. 

Now suppose that a is essentially self-dual of proper orthogonal type, so that there is a 
nondegenerate bilinear form G on V such that cr(G) C GSO(G). By Lemma 16.61 there is a 
3-space W C f\ 2 V such that p(GSO(G)) preserves W . In particular, W is G-invariant, and 
the reverse implication is proved. □ 

Proof of Theorem \6.5[ The representation A 2 cr is essentially self-dual. Thus if it has any 
proper nonzero G-invariant subspace, it necessarily has such a subspace of dimension at 
most 3. The proof follows from Propositions 16.7 1 16.91 and 16.101 □ 

Remark 6.11. Using the above theorem it is possible to explain the seemingly strange 
exception in (iii)(a) of Theorem 11.11 Let H be a subgroup of index 2 in a group G, and 
let (r, W) be a two-dimensional irreducible (£-adic) representation of H. Let a := As(r) 
be the corresponding 4-dimensional representation of G as in Section 12.21 where T" acts 
via G/H which switches the factors of W <8> W . Assume that a is irreducible. Since r is 
2-dimensional, there is a symplectic form S on W that r preserves up to similitudes. It is 
easy to see that a preserves S (g> S on W ® W up to similitudes. In fact, a is an essentially 
self-dual representation of improper orthogonal type. Further, one can see that r is dihedral 
if and only if a has a nontrivial quadratic self- twist. By the above theorem, one concludes 
that A 2 er is reducible if and only if r is dihedral. 
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